Reinforcement Learning: Model-based

Reinforcement learning (RL) refers to a wide range of dierent learning algorithms for improving a behavioral policy on the basis of numerical reward signals that serve as feedback. In its basic form, reinforcement learning bears striking resemblance to ‘operant conditioning’ in psychology and animal learning: actions that are rewarded tend to occur more frequently; actions that are punished are less likely to be repeated. Because of its simplicity, reinforcement learning is applicable to an incredibly wide range of dierent problems. Unfortunately, also because of its simplicity, RL often exhibits extremely slow learning in complex problems. An ongoing challenge in the field of machine learning is developing learning algorithms that share the generality and elegance of RL, but learn in a more ‘intelligent’ and sophisticated manner. This note considers one particular approach to extending the capabilities of RL algorithms, known as model-based reinforcement learning. Model-based RL is also of significant interest to researchers in cognitive science, as it is believed to more closely correspond to the type of learning that humans routinely demonstrate. Inmodel-based RL, as in the basic model-free approach, the primary goal of learning is the improvement of a behavioral policy in order to maximize a numerical reward signal. However, the approach differs frommodel-free RL in that simultaneously, the agent attempts to learn a model of its environment. Having such a model is a useful entity: it allows the agent to predict the consequences of actions before they are taken, allowing the agent to generate virtual experience, as well as perform mental search through a problem space to locate an ecient solution. Thus, model-based RL integrates both learning on the basis of past experience, and planning future actions. Further, a model of the environment is not directly tied to the task one is currently performing. For example, consider a mouse learning to navigate a maze to find a food reward. While performing this task, the mouse could acquire a mental representation of the structure of the maze. If the mouse’s goals change, for example, from finding food to finding a safe place to hide, the previously learned policy (the sequence of turns to locate the food reward) is no help, but the learned model of the maze structure could still aid the mouse in achieving its new goal. The downside ofmodel-based RL is that acquiring amodel in the first place adds to the overall